1. Field of the Invention
The present invention is generally related to expert forecasting and more particularly related to predicting the accuracy of expert forecasts based on a corpus of prior expert forecasts.
2. Related Art
The conventional disciplines of natural language processing, expert opinion analysis and to some degree portfolio selection are related to the problem of predicting the accuracy of an expert forecast based on a plurality of prior forecasts by the same expert. A summary of these disciplines and the problems associated with their use in predicting the accuracy of expert forecasts is provided below.
Natural Language Processing
Conventional opinion detection from a natural language document is carried out through a multi-pass natural language process that includes (a) statistically analyzing a sentence to separate factual content from opinion content; (b) re-analyzing opinion content as positive/neutral/negative using a second classification process; and (c) additional processing is performed to conform the opinion statement to a template in order to describe the subject and object of the opinion statement, the intensity of the opinion, as well as other application-specific data. Such an approach is necessary when the goal is to recognize opinions in a body of text.
Conventional systems typically use Bayesian classifiers to determine the sense in which a word is used in the body of text. Additionally, techniques for automatically distinguishing between fact and opinion content at the sentence level have been employed. Solutions for detecting polarity have also been used, although these solutions require predetermined sets of words having known polarity against which words in a sentence are compared. While conventional solutions for detecting polarity can determine if a given sentence has positive or negative polarity, they fail to determine whether the positive or negative polarity supported or opposed the ultimate opinion of the expert.
Expert Opinion Analysis
Conventional expert opinion analysis is typically associated with equities trading and is usually applied to cast the opinion in terms of a three or five point scale of recommendations (e.g., buy/sell/hold). Conventional expert opinion analysis is also applied to determine the expert's estimate of a company's earnings for the next quarter to be reported. The ability of the expert is then measured in terms of the error in the forecast relative to the actual reported result, and the degree to which the expert is considered to have delivered useful information is measured in terms of the market impact of the opinion, generally on the day that the opinion was disseminated.
Many problems exist within conventional expert opinion analysis. One such problem is that the analysis often is expressed in the context of the magnitude of error in the opinion relative to the magnitude of error of other experts. Other analysis techniques consider the timing of revisions to forecasts and assume that experts who revise forecasts earlier have a superior ability to discover and analyze information. These types of analysis fail to determine whether or not the expert opinion is accurate relative to the target subject but instead focus on relative comparison to other experts.
Other types of expert opinion analysis rely on the assumption that past performance forecasts future results such that experts with the greatest prior success in forecasting the behavior of a target subject continue to outperform in future recommendations. The converse assumption is also employed for underperformance. This type of analysis is also based on relative comparison and fails to consider the accuracy of an expert opinion relative to the target subject.
Still other types of expert opinion analysis rely on predetermined data to perform the analysis, do not make any predictions about the target subject of the opinion, do not include any temporal considerations regarding the underlying opinion data, rely on the expert to determine the weight of the expert opinion and also ultimately analyze an expert opinion based on factors as they relate to other experts. Conventional expert opinion analysis therefore fails to dynamically consider the most current opinion data from the expert, does not provide actionable information about the target subject, and fails to independently analyze the expert and instead only provides a relative comparison.
Another problem with conventional expert opinion analysis is demonstrated by the convergence/divergence technique. This conventional technique as applied to investments provides that divergence of opinion among experts correlates negatively with investment returns. In contrast, when experts agree with a positive outlook, investment returns are generally negative as equities prove to be overvalued. Similarly, when experts are collectively negative, returns tend to be positive. This technique suffers from the inability to analyze a single expert opinion because it requires a collective analysis.
Portfolio Selection
Portfolio selection refers to taking action based on an expert opinion analysis. However, the conventional techniques for identifying and analyzing expert opinions discussed above fail to also describe any techniques for taking action based on the substantive content of the opinion.
Some attempts at solutions for portfolio selection have been made as applied to allocation of funds among all possible stocks in a given market. Unfortunately, these attempts suffer from a necessary initial assumption of an equal-weighted starting portfolio where all stocks in the market are continuously held and capital is reallocated daily based on prior performance. A significant drawback of these conventional solutions is that they prevent withdrawing from a poor market, prevent shorting, and they impose substantial real-world trading costs. For example, some solutions require that there must be as many trades each day as there are stocks in the market and the resulting trading costs could easily exceed returns. Other proposed solutions attempt to capture side information, which is defined as information that is available to an investor that is independent of the price vector for each stock and is used as input to adjust and update the portfolio.
Other proposed alternative techniques employ a multiplicative update rule that uses a single tunable learning parameter and a vector-space distance function to control the changes between portfolios from day to day. These typically demonstrate improved returns over prior universal portfolio algorithms, with or without the use of margin borrowing and with or without the use of side information as an input which can improve algorithmic trading results. However, these systems generally behave identically to prior systems, with slightly improved rules used to make trading decisions.
Some of the significant drawbacks of these conventional systems are that to the extent expert opinions are used, these conventional portfolio selection techniques require predetermined data to perform the analysis. The required predetermined data includes a source of machine-readable representations of expert opinions for the various stocks or other instruments in the portfolio. Additionally, these techniques require access to an expert's portfolio allocation weights, which is generally not available for expert opinions. When the expert's portfolio allocation weights are not available, these conventional portfolio selection techniques cannot be used. Another very significant drawback of these conventional solutions is that they fail to allow for contrary actions in a portfolio but instead require the re-allocation of all portfolio funds, even when such a re-allocation would result in a predicted loss.
Some conventional solutions include artificial intelligence and technical analysis of stock prices. These portfolio selection solutions create a two-stage decision system in which a variety of technical analysis formulas commonly used by human traders are used to filter an initial universe of stocks down to a smaller list from which stocks to be bought or sold are determined based on a set of predetermined rules. These solutions fail to include human expert knowledge in the selection process but rather make selections based on historic stock price changes independent of the stock being analyzed by an expert or allocated a relative weight in a portfolio.
Other conventional solutions propose the use of multiple systems including a knowledge database and a set of rules to provide a recommendation and employ techniques for making a single decision from conflicting recommendations from the multiple systems. These solutions use a bidding system whereby the multiple systems conduct an auction to present their recommendations to a human being and that person provides positive or negative feedback regarding prior decisions from the expert. Although these solutions allow the multiple systems to modify the relative weights of their rule sets in response to feedback, they fail to analyze and employ an expert opinion to determine portfolio selection. A particular disadvantage of this approach is that feedback must be given by human beings. This approach also does not determine a best solution, but instead where the multiple systems agree, this approach always recommends inclusion.
Other conventional solutions approach the problem of portfolio selection by introducing technical parameters such as trends in prior stock prices to the selection of a constantly rebalanced portfolio. These solutions attempt to identify correlations between stocks so that when a first stock is correlated to a second stock that has recently increased, funds are reallocated from the second stock to the first stock to capitalize on the expected correlative increase in the first stock. These techniques require correlations to be computed over all pairs of stocks within the universe of available equities, making these approaches highly computationally expensive. Although these types of conventional solutions for portfolio selection have achieved some success, they fail to analyze expert opinions and incorporate those opinions into the portfolio selection process.
Therefore, give the state of the prior art, what is needed is a system and method that overcomes the significant problems found in the conventional systems as described above.